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Abstract: This paper studies a scheduling control problem for a single-server multiclass 
queueing network in heavy traffic, operating in a changing environment. The changing 
environment is modeled as a finite state Markov process that modulates the arrival and 
service rates in the system. Various cases are considered: fast changing environment, fixed 
environment and slow changing environment. In each of the cases, using weak convergence 
analysis, in particular functional limit theorems for renewal processes and ergodic Markov 
processes, it is shown that an appropriate "averaged" version of the classical c/i-policy 
(the priority policy that favors classes with higher values of the product of holding cost 
c and service rate fi) is asymptotically optimal for an infinite horizon discounted cost 
criterion. 
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1. Introduction 

Heavy traffic modeling has a long history. For a small sample of some recent works on heavy 
traffic analysis, we refer the reader to \TT\ [T^l I15j. a comprehensive list of references can be 
found in (9J [15] . The heavy traffic formulation provides tractable approximations for complex 
queueing systems that capture broad qualitative features of the networks. While most works 
in the literature deal with fixed (or internal network-state-dependent) rates, with the advent 
of modern wireless networks there is an explosion of research on models for networks where 
external factors such as meteorological variables (temperature, humidity, etc) can affect the 
transmission rates to and from the servers (see [3] and references therein). The current paper 
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deals with the simplest such setting of a multiclass network, namely a single server multiclass 
queueing system in heavy traffic, where the arrival and services fluctuate according to an 
environment process. In the classical constant rate setting this model has been studied in |13j 
in the conventional heavy traffic regime. The queueing system consists of a single server which 
can process K different classes of jobs (K > 1). The K classes represent different types of jobs 
(voice traffic, data traffic etc.), and they have different arrival and service rate functions. We 
assume that these functions are modulated by a finite state Markov process that represents 
the background environment. The arrival and service rates satisfy a suitable heavy traffic 
assumption which, loosely speaking, says that the network capacity and service requirements 
are balanced in the long run. We consider a scheduling control problem, where the controller 
decides (dynamically, at each time point t > 0) which class of jobs should the server process 
so as to minimize an infinite horizon discounted cost function, which involves a linear holding 
cost per job per unit time for each of the K different classes. The jump rates of the modulating 
Markov process are modeled by a scaling parameter v. The value of v can vary from negative 
to positive - the larger the value of v, the faster the environment changes - and we consider 
three distinct regimes for v (see (3.5)). We show that under different values of v, with suitably 
scaling, queue length processes stabilize leading to different diffusion approximations. Main 
result of the paper shows that, in each of these regimes, an asymptotically optimal scheduling 
policy is a variation of the classical and intuitively appealing c/x-rule. Classical c/x-rule [12J is a 
simple priority policy, where the priority is always given to the class (of nonempty queue) with 
the highest c//-value where c and ^ represent the holding cost and the service rate of that class, 
respectively. It is well known that c//-rule is optimal in many cases, see \10\ [T3] and references 
therein. In our model, the arrival rates change randomly according to the background Markov 
process, and it is far from clear if the classical c^-rule would be optimal. In addition, in one 
of the regimes we consider, the service rates are allowed to be modulated by the random 
environment as well, which means that the order of the c/i-values keeps changing according 
to the environment. In particular, in situations where the environment process is not directly 
observable and one only knows (or can estimate) its statistical properties, the classical c/i- 
rule cannot be implemented. We propose a modified c/x*-rule, where \x* represents the average 
service rate of each class and the average is taken with respect to the stationary distribution 
of the environment process. Under the c/i*-rule, the server always processes jobs from the 
nonempty queue with the highest C/U* -value. We show that, under an appropriate heavy traffic 
condition and a suitable scaling, the cfi* -rule is asymptotically optimal for the chosen cost 
functional for this environment-dependent queueing model, in each of the three regimes for v. 



As is common practice in heavy traffic analysis, we consider a sequence of networks, indexed 
by n > 1, each of which is a single server multiclass network in a random environment (see 
Fig. 0. In our formulation, the environment process operates at a time-scale of n u , where 
the scaling parameter v determines the rate of fluctuations of the environment process. When 
v > (Case 1), the background environment is changing faster than typical arrivals and services 
(which are 0(1)), while the reverse is true when —1 < v < (Case 3). If v = (Case 2), the 
variation rates of the background process and the arrival and service processes are both 0(1). 
The scaling that stabilizes the processes is different in each of the three cases: When v > 0, 
the space is scaling down by the usual factor of na, and when 1 < v < 0, the scaling factor 
becomes n~2~ (time is accelerated by a factor of n in all cases). In each of the cases, we show, 
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using functional central limit theorems for renewal processes and ergodic Markov processes, 
that the 'driving process' converges to a Brownian motion (the variance parameter of the 
Brownian motion is different in the three regimes, c.f. Lemma [5.3| ). The application of the first 
limit theorem is quite common in heavy traffic theory, however the use of limit theorems for 
ergodic Markov processes in optimal scheduling problems is less standard. We then formulate 
a Brownian control problem (BCP), which is a formal approximation of the queueing network 
control problem, where the driving process is replaced by a Brownian motion. This BCP is 
explicitly solvable and an optimal solution of the BCP is such that, the optimal state process 
(formal approximation of the scaled queue-length process) is zero for all the classes except for 
the one with the lowest value of the corresponding c/i* quantity. We use this insight gained 
from the solution of the BCP to propose a priority policy (see Definition pOT) , which prioritizes 
queues based on the c/x* value of the class. Our main result is Theorem |3.1[ which shows that, 
under conditions, this c[l* priority policy is asymptotically optimal for the queueing control 
problem, in each of the three regimes. 



As noted above, we consider three different regimes for v. (1) v > 0; (2) v = 0; (3) v £ (—1, 0). 
Additionally, Case 1 is further subdivided into: (a) v > 1/2; and (b) v £ (0,1/2]. For cases 
1(b), 2 and 3 we take the service rates to be constant (i.e. only the arrival rates are environment 
dependent). For case 1(a), both the service and arrival rates are allowed to be environment 
dependent. The key result that allows the treatment of environment dependent service rates 



is Lemma 5.2, where the condition u > 1/2 is crucially used. Although we leave the case 
when v < 1/2 and service rates are environment dependent as an open problem, numerical 
experiments in Section [3] suggest that even in these settings cpL* priority policy may be suitable. 



The paper is organized as follows. Section [2] describes the model, introduces the key as- 
sumptions and presents the main scheduling control problem. In Section [3j we introduce the 
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cost criterion and state the main result of the paper, namely Theorem 3.1 (see also Theorem 



4.1 ). Section [4] introduces the Brownian control problem associated with the scheduling control 



problem from Section[3| Section[5]is devoted to the proofs of Theorems 3.1 and 4.1 Finally, the 
appendix collects proofs of some auxiliary results and findings of our numerical experiments. 



The following notation will be used. Denote the set of positive integers by N. For K G N, 
let M. K be the i^-dimensional Euclidean space. Define M®q = {(xi,...,x K ) : x { > 0,i = 
1,2, ... , K}. Vectors are understood to be column vectors. For a K-dimensional vector x, de- 
note by diag(x) the K xK diagonal matrix whose diagonal entries are given by the components 
of x. For i = 1, . . . , K, Xi will denote the i th component of x. The transpose of x is denoted by 
x' , and the inner product of two i'T-dimensional vectors x and y will be denoted by x ■ y. Let | • | 

be a norm on WL K given by \x\ = \J^2d = i x\. For a matrix M, denote by Mjj the (i,j) th entry 
of M. Let D([0,oo) : WL K ) denote the space of functions that are right continuous with left 
limits (RCLL) defined from [0, 00) to H K with the usual Skorohod topology, and C([0, 00), H K ) 
the class of continuous functions from [0, 00) to M. K with the local uniform topology. Conver- 
gence of random variables X n to X in distribution will be denoted as X n =^ X , and weak 
convergence of probability measures \x n to \i will also be denoted as fi n =4> [i. A sequence of 
"D([0,oo) : M^)-valued random variables {A n } is said to be C-tight if and only if the mea- 
sures induced by {X n } on (D([0, 00) : M. K ), B(V([0, 00) : E ))) form a tight sequence and any 
weak limit of the sequence is supported on C([0, 00) : M. K ). For a stochastic process Y, we will 
use the notation Y{i) and Yt interchangeably, and we let AY(t) = Y(t) — Y(t—), t > 0. For 
semimartingales X, Y £ 2?([0,oo) : M), we denote by [X, Y] and (X,Y) the quadratic covari- 
ation and predictable quadratic covariation of X and Y, respectively. For a metric space U, 
let ^(U) be the collection of all probability measures on U and BM{U) the class of bounded 
measurable functions from U to M. For /j, G V(U) and / E BM(U), let = f v f(x)(i(dx). 
For / = (/i,..., IlY, where ft G BM(U), i = 1, 2, L, we write 

M(/) = (M/i),...,M^))'- 

A countable set will be endowed with the discrete topology. Finally, we denote generic positive 
constants by 01, 02, . . ., the values of which may change from one proof to another. 



2. Single-server multiclass queueing system 



2.1. Network model 



Consider a queueing system consisting of a single server which is shared by K G N parallel 
classes of customers as depicted in Figure [Tj Consider a sequence of such systems indexed by 
n G N. Let (Q n , F n , P n , {-7T}t>o) be a filtered probability space satisfying the usual conditions. 
All the random variables and stochastic processes for the n th system are assumed to defined 
on this space. For the n th system, we define a {J-" t n }-Markov process {Y™ := Y n {n l 't) : t > 0}, 
where i/£l and Y n is a {JFt/ n v} Markov process with state space L:={l,2,...,L},a unique 
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stationary distribution 7r™ := {-zr", ^ , . . . , vr£}, and infinitesimal generator (rate matrix) Q n 
which converges to the rate matrix Q of an ergodic Markov process. Note that as n — > oo, 
ir n converges to the unique stationary distribution it of this Markov process. The expectation 
operator with respect to P n will be denoted by E n , but frequently we will suppress n from the 
notation. We will make the following uniform ergodicity assumption. 

Assumption 2.1. For some R G (0,oo) and p G (0, 1) 



supmax|P n -*(i, j) -7if I < Rp\ t > 0, 
where P n '*(i, j) = P n (Y n (t) = j | Y n (0) = i) . 

As shown in Figure [TJ there are K classes of external arrivals. For i G IK := {1,2,..., K}, 
customers of Class i arrive from outside to buffer i and upon completion of service, they leave 
the system. For t > 0, denote by (t) the number of customers of Class i arriving from the 
outside by time t and S™(t) the number of customers of Class i completing service by time t 
if the server is continuously working on Class i customers during [0, t]. We assume A™ and 5™ 
are Poisson processes with intensities modulated by Markov process Y n . More precisely, for 
i G K, let A?, fif : L ->• M+. Then A™ and fip , i G K can be described as follows: For t > 0, 

^(*) = , 5?(t) = ^ ^(17)*) , (2-1) 

where iV"^ and A^"^ are independent unit rate Poisson processes. Here A", are the arrival and 
service rate functions, respectively. The processes Y n and N™i, N%j,i, j G K are assumed to be 
mutually independent. Also, we assume that A?(t) - Jo X 7 l(Y n (s))ds is a { JT}t>o martingale. 



An analogous assumption on 5" will be introduced in Section 2.2 (see Condition 2.1 (hi)). Let 
X n = (A™, . . . , X\)' and p n = (/x™, . . . , p\)' ■ We make the following convergence assumptions 
on A" , p n . Dehne the "averaged" arrival and service rates as follows: 

\n,* „n(\n\ ,.n,* _n/,,n\ 

X = vr (A ), p = 7r [fx ). 

Assumption 2.2. 

(i) There exists a function X : L — > IR®^" such that as n — )• oo, X n (y) — > X(y), for all y G L. 

(ii) There exists p* G R®q such that as n — ^ oo, — > fj,*. 

Let A* = vr(A). Assumption (i) in particular says that the arrival rates are 0(1) and 
lim n _>oo A n '* = A*. Assumption on the service rates is somewhat weaker in that it only re- 
quires that the "averaged" service rates are convergent. Additional conditions on X n and p n 



(heavy traffic condition) are formulated in Assumption 2.3 at the end of Section 2.2 
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In the n th network, the scheduling policy is described by a service allocation process 

T n (t) = (T?(t),T?(t),...,TZ(t))',t>0, 

where for igK, Tp(t) denotes the total time used to serve customers of Class % during [0, t]. 
The idle time process is defined as follows: 

K 

I n (t) = t-J2T t n (t),t>0, (2.2) 

8=1 

which gives the total idle time of the server during [0, t]. Denote by Q n the if-dimensional state 
process, i.e., the queue length process (including the job being served). Then the evolution of 
the queueing system can be described as follows: For ieK, 

Qi(t) = Ki (/ A?(y " )ds ) " N ^ (fi ti(Y?)<n?(*)\ » * > o- 

Here, for simplicity we assume that at time the system is empty. 

The processes T n and I n are required to satisfy the following conditions: For i G IK, 

T"(-) is continuous and nondecreasing, and T™(0) = 0, 

I n {-) is continuous and nondecreasing, and I n (0) = 0, (2-3) 

Q?(t) > 0, for all t > 0. 



As a consequence of the conditions in (2.3), we have the following Lipschitz property of T n : 
For % G K, 

T™(-) is uniformly Lipschitz continuous with Lipschitz constant bounded by 1. (2.4) 

Therefore, T™(-),z G K, is almost everywhere differentiable. Denoting its derivative by T™(-), 
we have 

2f(t)= PiTWds, t>0. (2.5) 



J o 

We make the following additional assumptions on {T n }. Define a sequence of nondecreasing 
random times }fceNu{o}- Let Tq = 0, and for k G N, r^ +1 be the first time after when 
there is either an arrival or a service completion. 

Condition 2.1. 

(i) For allkenu {0},f n (t) = T n (r") for all t G [r£, t™ +1 ). 

(ii) The process {T n (t)}t>o is {J 7 ™} -progressively measurable. 

(iii) AT" (j^2(Y s n )dTt(s)) - titf(Y?)dT?{s) zs a {J?} t > martmgale. 
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Part (i) of the condition says T n does not change values between two successive changes 
of the state of the system. Part (ii) is a natural nonanticipativity property of T n and the 



martingale condition in part (iii) is analogous to the one imposed on Af in Section 2.1. We call 



T n satisfying (2.3) and Condition 2.1 an admissible control policy and denote the collection of 



all such T n by A n . 



Using (2.5), we can write Q™,i G IC in the following way: For t > 0, 



Q?(t) = (Jl \?(Y s n )ds) - N? ti (Jf tf(Y s n )Tr(s)ds) . (2.6) 

Finally, we write N% = (N^, . . . , N^ K )' and iV" = (JV£, . . . , N^ K )' . 

We now define the fluid and diffusion scaled processes. Roughly speaking, the fluid-scaled 
processes are defined by accelerating time by n and scaling down space by the same factor, 
and the diffusion-scaled processes are defined by accelerating time by n and scaling down space 



(after an appropriate centering) by n a for some a G (0, 1). Choice of a is specified in (3.5). 
Fluid scaling: For t > 0, 



n 



f n ( t ) = r(t) 



n 

I n (nt) 



n 



(2.7) 



n 



n 



Diffusion scaling: For t > 0, 



ii L 



From the above scalings and (2.6) we see, for i € K and t > 



and 



jit 



lit 



lit 



lit 



Qf(t) = N% J XUY s n )dsj - JV£ (- J tf{Y?)T?{s)ds 

+ \ r ( X ?( Y s n ) ~ tf(Y s n )Tr(s)) ds. 
n Jo v ' 



(21 



(2.9) 



(2.10) 



We now present our main heavy traffic condition. 



Assumption 2.3 (Heavy traffic condition). Let X* and /i* be as in Assumption 2.2. Then 

K 



i=l 



(2.11) 
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and furthermore, for each i G K, there exists bi G K suc/i i/iai as n — > oo ; 



-V - ^ ) 6i- ( 2 - 12 ) 



3. Objective and main results 



For our optimization criterion we consider an expected infinite horizon discounted linear hold- 
ing cost. More precisely, for the n th network, the cost function associated with the control 
policy T n is defined as follows: 



j n (T n )=E[ e-^(c-Q n (t))dt) , (3.1 



where 7 G (0, 00) is the "discount factor" and c := (c\, C2, . . . , ck) £ M®^" is the vector of 
"holding costs" for the K buffers. 

The goal of this work is to find a sequence of admissible control policies {T n : n G N} such 
that it achieves asymptotic optimality in the sense that 



lim J n (f n ) = inf liminf J n (T n ), (3.2) 

n— >co n— >co 

where the infimum is taken over all admissible control policies {T n }. 

For single server multiclass queueing systems (with constant rates), one attractive policy is 
the well-known c/i rule, where c and fi stand for holding cost and service rate, respectively. 
Under cfi rule, the server always selects jobs from the nonempty queue with the largest cfi 
values. In this work, we generalize the c\x rule to the Markov modulated setting and show that, 
under conditions, an "averaged" cfi* rule is asymptotically optimal, where fj,* is the "averaged" 



service rate introduced in Assumption 2.2 (ii). Our precise result is given in Theorem 3.1 
below. We begin with the following definition. Let (cr(l), <r(2), . . . ,a(K)) be a permutation of 
(1,2,..., K) such that 

C a (l)fC(l) ^ C <r(2)tC(2) > ■ > c a{K)fC(K)- ( 3 - 3 ) 

Definition 3.1 (c/d* rule). For i G K, let 

T "(i)(*) = J o 1 {Q^ (l) ( S )>o}n}= 1 1 l { Qn ( . ) ( s)= o}rfs- (3.4) 



Note that T n defines a priority policy in which the server always gives service priority to 
the nonempty queue with the largest value of Cj/i|, iel. 



Theorem 3.1 proves the asymptotic optimality of {T n }. Furthermore, in Theorem 4.1 



we 



will characterize the limit of J n (T n ) in terms of the solution of a Brownian control problem 
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which is introduced in Section [4} Recall that the background environment Markov process 
modulating the network is {Y n {t) = Y n (n"t) : t > 0}. Consider the following three regimes for 
v and a. 



Case 1. 



(a) v > -, a 



(b) 0<u< 1/2, a = -, and /x"(y) = //*•*, for all y £ L. 



(3.5) 



Case 2. v = 0, a 



and // n (y) = //*'*, for all y £ L. 



Case 3. - 1 < < 0, a = ^ and /U n (y) = //*'*, for all y € L. 

Case 1 considers a situation where the jump rates of Y n are much higher than the interarrival 
and service rates. For Cases 1(b), 2 and 3, the service rate \i n is independent of Y n . In Case 
2, Y n has jump rates of the same order as the arrival and service rates, while in Case 3, Y n is 
changing slowly in comparison with the arrival and service processes. 



Throughout this work Assumptions 2.1 2.2 and 2.3 will be taken to hold and will not be 
noted explicitly in statements of results. 



Theorem 3.1. Let {T™} be as in Definition 3.1 Then for each of the three cases in (3.5), 

lim J n {T n ) = inf liminf J n (T n ), 

n— >oo n— >oo 

where the infimum is taken over all admissible control policies {T n }. 

Remark 3.1. Note that in Cases 1(b), 2 and 3 the proposed control policy reduces to a standard 
c^-rule. Thus Theorem \3. 1\ shows that in these regimes, the classical en rule to be optimal under 
conditions that in particular require an 'averaged' form of heavy traffic condition (Assumption 



2.3) to hold. 



Remark 3.2. One may conjecture that Theorem 3.1 is true even when the condition that fi n 
is constant is dropped from Cases 1(b), 2 and 3. The key difficulty in treating these cases is 
in the proof of results analogous to Lemma 5.2 . Although we are unable to treat these cases, 
simulation results below suggest that the cfi* rule performs better than the dynamic c/i rule in 
these settings. 



Example 3.1. Consider the special case of the model in Section \2A\ with, 

K = {1,2}, L = {1,2}, vr 
For the n th system, let for y £ L, 

A?(y) 

Then 



-,-]• <■- (20.2.1). 



by/n 2 y/rt 2 5y/n 



3y Sy 
2 y7n' 



1 + ^=, lh 



n 



1 ^ ^1 

i u \ n >* i n,* 



5 5 
2 + 
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and 

k q x. 

\* — i ,,* — \* — ,,* — 

A i — J-i Mi — 2' A 2 — 2' ^ 2 ~ o' 



Clearly Assumption 2.3 is satisfied with a = 1/2 and b = (—2/5,-4/5)'. Note that for all 
n £ N ; ci/ij' < C2H2'* an d so buffer 2 gets service priority under the c/i* ra/e. However, 
for the dynamic c/i rule, the priority changes according to the state of the modulating Markov 
process, since for n > 2, 

cim?(1) > c 2 ^(l), cim?(2) < c 2 ^(2). (3.6) 

More precisely, in the dynamic c/i rule, when Y n (t) = 1, customers of Class 1 have priority to 
be served, while when Y n {t) = 2, priority should be given to Class 2 customers. The numerical 
study given below compares the performance ofcfi* rule with the dynamic cjj, rule. The following 
table summarizes the simulation results for different scalings. We consider the cost criterion 
( |3.1[ ) with 7 = 2. Let T = 100. We simulate 10 sample paths of Q n (t), which are denoted as 
(<7l sC0> l2s(t))' s = 1, 2 . . . , 10, for each scaling case at discrete times t = 0.1, 0.2, . . . , T. We 
then calculate the discounted cost as follows: 



10 T 10 T 

c ° st = t EE + 1 E E 

s=l (=0.1 s=l£=Q.l 

We find in our numerical results that the discounted cost is always smaller under the cfi* rule. 



n 


V 


Q 


Cost (c\i* rule) 


Cost (dynamic c/i rule) 


25 


1 


1/2 


52.70 


55.97 


100 


1 


1/2 


60.18 


61.87 


25 


2/3 


1/2 


72.66 


78.57 


100 


2/3 


1/2 


75.07 


77.13 


25 


1/3 


1/2 


63.93 


65.04 


100 


1/3 


1/2 


81.60 


84.33 


25 





1/2 


64.91 


69.69 


100 





1/2 


68.22 


75.52 


25 


-1/3 


1/2 


79.58 


81.95 


100 


-1/3 


1/2 


55.63 


58.06 


25 


-1/3 


2/3 


41.25 


44.20 


100 


-1/3 


2/3 


25.80 


27.23 



Also note that only the first four rows of the table correspond to Case 1(a). The remaining 
rows, although consistently show that c/i* rule outperforms the dynamic cfi rule, correspond to 
regimes not covered by our results. We also plot the cost at each discrete time for the average 
sample path (of the 10 sample paths) for a range of values of v when n = 100. The figures can 
be found in Appendix\C\ 
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4. Brownian control problem 



Our control policy {T™} is motivated by the solution of a Brownian control problem (BCP). In 
this section we introduce this BCP. Roughly speaking, the BCP is obtained by taking a formal 
limit, as n — > oo, in the equation governing the queue length evolution for the n th system. 
Recall the evolution equations for Q n from (2.10). The last term on the right side of (2.10) 
can be rewritten as 



1 



nt 



A' 1 ' 



M, 



Mi 



, | n,* I— a 



+ 



nt 



wo?) - \r) ds 



X*t 



nt 



T?(t) 



Mi 



T?(s)ds. 



Let 



= N r a 



nt 



+ 



1 



n Jo 

nt 



XUY s n )ds)-N ; 



nt 



tf(Y s n )T?(s)ds 



n 



(A?(n n ) 

, n,* 



xr) ds 



nt 



{tf(Y s n )-vr)Tr(s)ds 



(4.1) 



i n,* i— a 



A 



Mi 



M* 



and 



Then for i G K and i > 0, 



n 



l-Q 



Aft 
M? 



5T(t) 



Q?(i)=xr(i)+/iri-(*)- 



We now formally take the limit as n — > oo. Write X n = (X" 
Lemma |5.3| it is shown that, under conditions, as n — > oo, 

X n =>■ X, 



(4.2) 
(4.3) 

,r K y. in 

(4.4) 



where X is a X-dimensional Brownian motion with drift $ and covariance matrix E, where i? 
and E are given explicitly in Lemma 5.3 In particular the value of E is different in the three 
cases considered in (3.5). Also, as n — > oo, // n '* — > (jl* . Although in general f) n will not converge, 
formally taking limit in (4.3), as n — > oo, we arrive at the following BCP. 

Definition 4.1 (Brownian control problem (BCP)). Let X be a K -dimensional Brownian mo- 
tion with drift ■& and covariance matrix E given on some filtered probability space (O, P, {J-f})- 
The BCP is to find an K -valued RCLL {.Ft} adapted stochastic process fj = (fji, 772, ■ • • , t\k)' 
which minimizes 



E 



e-^(c-Q(t))dt 



/BCP for Markov modulated parallel queues 



12 



subject to 



Qi(t) = Xi(t) + fJ,*fji(t) > 0, for alli eK,t> 0, 

K 

!{■) := ^J?7i(-) is nondecreasing and 1(0) = 0. 



(4.5) 



i=l 



Denote by A the collection of all M. K -valued RCLL {Ft} adapted stochastic process f\ that satisfy 



(4.5) 



Brownian control problems of the above form were first formulated by Harrison in [6j, and 
since then, many authors have used such control problems in the study of optimal scheduling 



for multiclass queuing networks in heavy traffic. The BCP in Definition 4.1 has an explicit 



solution given in Lemma 4.1 which can be proved by standard methods (see pQ and references 



therein). For completeness we provide the proof of Lemma 4.1 in Appendix |A[ 
Lemma 4.1. Define for t > 0, 



K 



no 



A' 



Xiit) 



0<s<t' — ' U 

— 1=1 r ' 



i=l 



and 



v: (K) (t) = w*(t)-^p^, n*(t) 



a(K) 

Then rj* is an optimal solution of the BCP. 



Xj(t) 



j = a(l),...,a{K-l). 



(4.6) 



Define for i € K, Q* by (4.5) with fji replaced by r\[. Then we have 

Q**(K)(t) = ti(K)W*(t) > 0, and Q*(t) = 0,j = a(l), ...,a(K- 1). (4.7) 



The optimal solution (4.7) suggests that the jobs should be always kept in the buffer with 



smallest value of Q/x*, i G K. This motivates the control policy defined in Definition |3.1| Define 

J* = E 



-^(c-Q*(t))dt 



From Lemma 



4.1 



we have J* = inf E ^ e 7< (c • Q(t))dt^, where the infimum is taken over 



all fj € .A. In fact, we have the following result, which is proved in the next section. 



Theorem 4.1. Let {T n }„>i be as in Definition 3.1. Then for each of the three cases in (3.5) 
lim^oo J n (T n ) = J*. 



5. Proofs of Theorems 13.11 and 14.11 



In this section we prove Theorems 3.1 and 4.1 . We begin with the following two lemmas, which 
play a crucial role in the proofs. The first lemma is a functional central limit theorem (FCLT) 
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for the sequence of Markov processes Y n . We first introduce some notation. Denote by Q n the 
infinitesimal generator of Y n , namely Q n : BM(h) — > BM{h) defined as 



(5.1) 



As in Assumption 2.1, denote by P n ' the transition kernel of Y n , namely, for y E ~L,A C 
L, P n '*(y, A) = F n (Y n (t) E A \ Y n (0) = y). For notational simplicity we denote this probability 
by ¥y(Y n (t) E A) and by K y the corresponding expectation operator. In particular for / E 

BM(L), P n '*(y ; /) = Define for t > and y E L, 



A?' C (2/) := Ej 



\ n (Y™)ds-tTT n (\ n ) . 



(5.2) 



Lemma 5.1. For y E L, limjj-^oo lim^oo A"' c (y) exists. Denote the limit by X(y). Let {?(n)} 
6e a nonnegative sequence such that ?(n) — )• oo as n -> oo. Define for t > and n > 1 



G n (i) 



1 



?(n)t 



(A"(y™) - \ n '*)ds. 



Then G n converges weakly to a K -dimensional Brownian motion with drift and covariance 
matrix A, where for i,j E K, 



A; 



L 



(Xi(y) - A*)A,(y) + (A,(y) - A*)A 4 (y) 7r(dy). 



Proof: From Assumption 2.1 there exists ai E (0, oo) such that for < t\ < ti < oo, 



K c (v) - K; c (v)\ < T l^' s (y, a-) - n n (x n )\ ds < ai R t P s d s = ai pio g ( P )(^ - ^). 



This shows that lim^oo A"' c (y) exists, and we denote this limit by X n (y). Note that X (y) 
/ °° E y (A n (y s n ) - vr n (A n )) ds. Next note that for y E L, 



Q»A"(y) = lim Q"A[ l ' c (y)= lim f /' V Q"P n ' s (j, A n )ds - t^ n (A n ) V ( 

V u jfeL jeL 



lim / V Q",P n ' s (j,A n )ds = lim 



j&L 



1 dP n 



(5.3) 



ds 



-(y,X n )ds = n n (X n )-X n (y). 



Noting that Q n converges to Q, it follows that (cf. [4, Theorem 4.2.5]), if Y n (0) has limit 
distribution v on L, then there exists an L- valued Markov process Y w ith initial distri butio n v 
and rate matrix Q such that Y n => Y. Therefore, by Assumption 2.2 (i), Assumption 
dominated convergence theorem, as n — > oo, 



2.1 



and 



A n (y) 



(A"(^ n ) - 7r n (A n )J ds -> jf E^A^-^A))^, 



/BCP for Markov modulated parallel queues 
This proves the first statement in the lemma, in fact 

X(y) = rE y (x(Y s )-7T(X))ds. 



14 



Following the proof of (5.3) one can check that QX(y) = vr(A) — X(y) for y G L, where Q is 
defined as in (5.1) with Q n replaced by Q. Next observe that for each n, 

M? := X n {Y t n ) - A n (y n ) - f Q n X n (Y s n )ds, t > 0, (5.4) 

J o 

is a = o~{Y™ : s < t} square integrable martingale. Furthermore 

sup \G n {t) - ? (n)-§M" fn J < ? (n)~3 sup \X n {Y n (q(n)t)) - A"(Y n (0))| -»■ 0. (5.5) 
<>o t>o 

Thus it suffices to show 
$(ti)~ 5 M^V \ converges weakly to a Brownian motion with drift and covariance matrix A. (5.6) 



Proof of (5.6) follows from classical functional central limit theorems for martingales. For 



completeness, we provide the details in Appendix [B] ■ 

Lemma 5.2. Let {f n } n >i, f n '■ L — > M, be a sequence of functions such that sup ngNj/gL \ f n (y)\ < 
oo. Define for t > and i£K, 



1 



$/ n (i) = — / (p(Y s n ) - mT t n (s)ds, 

v n Jo 

where f n '* = 7r n (/ n ). Suppose that v > 1/2. Then, there exists K\ G (0, oo) sitc/i £/ia£, for all 
t > 

E [ sup |$ /n (s)|l < 

_0<s<t II 

In particular, as n — >■ oo ; 



i/-l/2 ' 



(5.7) 



Proof: Note that for i > 0, 
-7= / ( W") " f n '*)T?{s)ds = — T / 



(fK") - f n '*)T?{s/n^)ds 



+ 



(f^-Dir^/n")^. 



For the last expression in (5.8), note that there exists a\ G (0, oo) such that for all t > 0, 



{f n {Y?)-P*)T?{s/n»)ds 



< 



a i 



(5.9) 
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n ' 2 



(f n (Y s n ) ~ r*)T?{s/n»)ds 



k=0 

\n u +H\-l 



+ 



k=0 



(f n (Y s n )-rn{Tr 



k-2 



rr 



ds. 



Next note that, there exists ai £ (0, oo) such that for 2 < k < Lra^ +1 tJ — 1, 

'k- 2" 



E sup 

V k<s<k+l 



rpn 



rpTl 



< 



sup 

fc-2<s<fc+l 



rr 



rv 



rr 



k-2 



n 1 ' 



(5.10) 



> < 



«2 



n 1 ' 



where the last inequalit y is a consequence of the fact that the arrival and service rates are 
0(1), and by Condition 2.1 (i), T n does not change values between two successive arrivals 
and/or service completions. Thus we can find 03 E (0, 00) such that for all t > 0, 

'k-2 



E I sup 

0<s<t 



1 K +1 sJ-i rk +i 
^+1 E 



n 2 



fc=o 



du 



< 



Define g n € BM(V) as 



a 3 t 



(5.11) 



g n (y) = / E y \f n (Y n (s))) — vr n (/™) ) ds 



From Assumption 2.1 and calculations as in the proof of Lemma 5.1 it follows that max^gL sup n \g n (y)\ < 
00. Also, 

M n (s) := g n (Y s n ) - g n (Y n ) + f (fW) - f n <*)du, s > 

Jo 

is a {£ t n } martingale, where gf = o{Y™ : s < t}. For < k < [n u+1 t\ - 1, let = 
M n (k + l)-M n (k).Then 



K+^J-i n _ 2 s fk+1 



n u+ 2 



k=0 
[n v+1 t\-l 



k=0 



k-2 



(f n (Y s n )-rnds 

9 n (Y k n +1 )-g n (Y k n )-C k 



/BCP for Markov modulated parallel queues 



16 



The last expression equals 

[n v+1 t\-l 



n 1 2 



rpn 



A=0 

K +1 tj-i 



A ' ^(n n + i)-7f(^)5 n (n n ) 



+ 



A=0 



fc- 3 



fc-2 



l ( f k)-—r E 



A=0 



fc-2 



(5.12) 

The first term in the above display is a telescoping sum and since {g 11 } is uniformly bounded, 
we can find 04 S (0, 00) such that for all t > 0, 



K +1 tj-i 



h4 £ 



n"' + 2 



fc=o 



fc-2 



g n (Y k n +1 )-TP 



fc-3 



< 



(5.13) 



Also, from (5.10), there exists 05 £ (0, 00) such that for all t > 0, 

3 \ ■ , r -. / k 2 



E I sup 

0<s<t 



k=0 



n" 



TP 



11 



9 n (Y k n ) 



< 



(5.14) 



Now we consider the last term in (5.12). 



For < fc < \n v+ H\ - 1, let §f = If (^#) By Condition ^ (ii), E = and 

so £j? is a martingale difference. Furthermore, is uniformly bounded in n and fc. Applying 
Doob's inequality, there exists qq G (0, 00) such that for i > 0, 



E I sup 

0<s<t 



|n" +1 sj-l 



n 2 



A=0 



fc-2 



n" 



'Ln" +1 tJ-l 



< 



2i/+l 



E 



E *r 



A=0 



fc-2 



71" 



Ln" +1 tJ-l 



(5.15) 



72 



2u+l 



A=0 



The result follows on combining (5.9), (5.11), and (5.13) - (5.15). 



The following lemma gives a functional law of large numbers (FLLN) and a FCLT. For a 
sequence of admissible policies {T n }, i£K and t > 0, let 



(5.16) 



Recall {Af } from equation plj). Write A n = . . , A n K )' and D n = (D™, . . . , Dk)' ■ Define 

the fluid scaled processes ^4 n (t) = n _1 A n (nt) and = n~ 1 D n (nt). Let t : [0, 00) — > [0, 00) 

be the identity map, and define for t > 0, 



T*(t) 



2^1 2^2 



X* 



t. 



(5.17) 



Finally, recall X™ defined in (4.1). 
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Lemma 5.3. Let {T n } be a sequence of admissible scheduling policies such that 

J({T n }) = liminf J n {T n ) < oo. (5.18) 

n— >oo 

Let {n'} be a subsequence of {n} along which the above liminf is attained. 

(i) The following FLLN holds: As n' — > oo 

(Q n ',A n ',D n ',f n ',I n ') (0,\*L,\*i,f*,0). 

(ii) The following FCLT holds: As n' — > oo 

X n ' => X, 

where X is a Brownian motion with drift i? = (n\b\, . . . , i^bx)' and covariance matrix 
£ which is given in the three regimes as follows: 

Case 1. S = diag(2X*). 
Case 2. S = diag(2X*) + A. 
Case 3. S = A. 

Proof: With abuse of notation, we relabel the subsequence {n'} once more as {n} and with 
this relabeling we have that 

lim J n (T n ) = J({T n }) < oo. (5.19) 

n— >-oo 

Part (i). Let i € K. From functional law of large numbers for Poisson processes, we have 

2V£(-) andiV^O^O- (5-20) 

Next note that 

- / (Arm - \^)d. = -ir / (a?(y?) - AT)*. 



Thus from Lemma |5. 11 we have that 

WttT) - Oto => 0, (5.21) 



o 



and thus by Assumption |2.2[ i) , 

1 I'" \UY s n )ds^\*L(-). (5.22) 
n Jo 

Using the random change of time theorem (see Lemma 3.14.1]), we have that 

i?(0=> W). (5.23) 
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In cases 1(b), 2 and 3 (i.e. when fi n (Y n ) = /U n '*), by Assumption |2.2[ii), we have that for t > 0, 



E ( sup 

\0<s<t 



< 



(5.24) 



Furthermore, in case 1(a) (i.e. when v > 1/2 and jj, n is modulated by Y n ), using Assumption 



2.2 



(5.24) and (5.20), we have that 



From (2.4) 



and 



ii) and Lemma 5.2 (see ( |5.7[ )), we have the same convergence result as above. Combining 

L>™ is C-tight. (5.25) 

TV- is C-tight. (5.26) 

(5.27) 
(5.28) 



We next note that from (2.9) and (2.2), 



Q n (t) = A n (t)-D n (t), 



K 



i=i 



From (5.23), (5.26) and (5.25), we now have 

(Q n , I n , A n , D n , f n ) is C-tight. 



(5.29) 



Let (Q, I, A, D, T) be a limit point, along a subsequence {n"} of {n}. By Skorohod representa- 
tion theorem, we can assume that this convergence holds almost surely, uniformly on compacts. 
From (5.19) and using Fatou's lemma, we have 



0= lim ^=J n "(T n ") >E 

n"— >oo v/77," 



E 



-it 



liminf/i-Q n (t) 
e" 7i [h ■ Q(t)] dt 



dt 



(5.30) 



Since h > and Q has continuous sample paths, we have Q = 0. By (5.23) and (5.27), 



(5.31) 



Also, from ( |5.20[ ) and ( |5.24[ ), we have that fi*T{. Therefore, 

A* 

TAt) = -^t, t > 0. 



(5.32) 



Finally, (5.32) and heavy traffic condition (2.11) imply I n =^ 0. This completes Part (i). 



We now consider part (ii). From (4.1) we have that 



nt 



n 



nt 



+ n-^ + ^Gf +1 (t)-n- a+1 2^"(t)+^*n 1 - a 



tf{Y?)T?{s)ds 

A™'* A! 



n ;* ,,=1 



(5.33) 



t. 
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We now treat the three cases separately. 

Case 1. From functional central limit theorem for Poisson processes, we have that 



n \K 
s,i)i=l 



K 

i=l) 



(5.34) 



where N a i and N s 



1, • • • K are 1-dimensional mutually independent standard Brownian 



motions. From (|5.22|), (|5.24|), and (|5.32|), we have 

tf(Y s n )iT(s)ds 



n 



K(Y?)ds -N? { 



1 

n ./n 



A" 



8=1 



(Wi)^i. 



(5.35) 



where Wi, i = 1, • • • K are 1-dimensional mutually independent Brownian motions with drift 
and va rian ce 2 A*. In case 1(a) (i.e. when v > 1/2, a = 1/2 and /u n is modulated by Y n ), by 



5.1 



and 



5.2 



we have n 2 Gf " +1 + <I> M 



0. In c ase 1(b) (i.e. when v > 0, a = 1/2 and 
Thus, using (2.12), {Xf)f =l converges 



5.1 



Lemmas 

fi n = fj, n '*), we have the same convergence by Lemma 

weakly to (Xi)fL l} where Xi, i = 1, • • ■ K are mutually independent Brownian motions with 
drift n*bi and covariance 2A*. 

Case 2. Here a = \ and v = 0. As in Case 1, we have the convergence in (5.35). Also note 
that <$>tf 



since = fi- 



Furthermore, by Lemma 



5.1 



we have 



G' 



W, 



(5.36) 



where W is a K- dimensional Brownian motion with drift and covariance matrix A. The result 
now follows on recalling that, Y n , N^, N™-,i, j G K are mutually independent. 
Case 3. Here a = and — 1 < v < 0. From Lemma 
to hold. Also, since a > g, we have 



5.1 



the convergence in (5.36) continues 



K(Y s n )ds 



Finally, ^ 



n Jo 

0. The result follows. 



N n - 

1 ' S,l 



n 



tf(Y?)T?{s)ds 



0. 



We recall below the definition and basic properties of the 1-dimensional Skorohod map. Let 
£> ([0,oo),R) ={i£ 2?([0,oo) : M) : i(0) > 0}. 

Definition 5.1 (1-dimensional Skorohod Problem (SP)). Let x G ^o([0, 00), M). A pair (z, y) € 
X>([0,oo) : E + ) x X>([0, 00) : M + ) is a solution of the Skorohod problem for x if the following 
hold. 



(i) For all t > 0, z(t) = x(t) + y(t) > 0. 

(ii) y satisfies the following: (a) y(0) = 0, (b) y(-) is nondecreasing, and (c) y(-) increases 
only when z(-) = 0, that is, J °° z(t)dy{t) = 0. 

We write z = T(x) and refer to the map T : Dq([0, 00), R) — > P([0,oo) : M+) as the Skorohod 
map. 



The following proposition summarizes some well known properties of the 1-dimensional SP. 



Proposition 5.1. 
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(i) Let x E Do([0, oo), R). Then there exists a unique solution (z,y) E 2?([0,oo) : IR+) x 
"D([0, oo) : R_|_) of the SP for x, which is given as follows: 

y(t) = — inf x(s), and z(t) = x(t) — inf x(s), t>0. 

0<s<t 0<s<t 

(ii) The Skorohod map T is Lipschitz continuous in the following sense: There exists k E 
(0, oo) such that for all t > and xi,x 2 E T>o([0, oo), M), 

sup |T(xi)(s) - r(x 2 )(s)| < k sup \xi(s) - x 2 (s)\. 

0<s<t 0<s<t 

(iii) Fix x E T> ([0, oo),R). Let (z,y) E £>([0,oo) : R+) x £>([0,oo) : R+) be such that 

(a) z(t) = x{t) + y{t) > 0, t > 0, 

(b) y(-) is nondecreasing with y(0) = 0. 
Then z{t) > T(x){t), t > 0. 



We next present two lemmas that will be used in the proofs of Theorems 3.1 and 4.1 Recall 



the control policy T n introduced in Definition 3.1 Lemma |5.4| says that the long run average 



of T n approaches T* and Lemma 5.5 shows that under T n , all the jobs are kept in buffer o~(K), 



i.e., the buffer with smallest value of Cifi*,i E K, asymptotically. 
Lemma 5.4. As n — > oo, T n =^ T* . 

Proof: In the proof, all the quantities, e.g. queue lengths, idle times etc., are considered 
under the control policy {T n }. 

Define, for n > 1, W n (t) = J2? =l ^W~, and let 

ft i 

*. W .EM.f^W.t^W +m ,5.37, 



Note that 



I n (t) = i- (nt - £| T?(ni) j = n 1 - £ l { ^ (s)=0)i6K} d S = n 1 "" jT l { ^ w=0} d.. (5.38) 

Clearly / n (0) = 0, /(•) is nondecreasing and increases only when W n {-) = 0. Therefore we have 
(see Definition HOT) , for t > 0, 



W n (t) = T[Y J ^)(t)- (5-39) 
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Using the Lipschitz property in Proposition 5.1 (ii), for t > 0, 



K 



sup W n (s) < ^ -n^r sup 



0<s<t 



i=1 Mi 0<s<t 



xr(s) 



(5.40) 



Fix i G K. Then from (|5.33|), we have for all t > 0, 



sup 

0<s<t 



< sup 

0<s<i 



N n ■ 

a,i 



+ sup 

0<s<t 



Mj " I n,* * 
Mi Mi 









+ sup 




( - f 


0<s<t 




K n Jo 



+ n a+ 2 sup <J? Mi (s) 



0<s<t 



_|_ n gU p 

0<s<t 



For any t > 0, we have 



max 



rat 



^(Y«)T™(«)^, 



o 



n 



nt 



o 



Ar(i; n )^ < max (A?(0 + M?(0)*- 

ZGL,n6N 



Let a± := max; g L„ g N(A™(/) + //"(/)). From Assumption 2.2 (i) and (ii) we have (without loss 

3. Using D( 

Arcc)^' 



of generality) that a\ < oo. Using Doob's inequality, we have for some 02 G (0, 00 

2 



E ( sup 

V0<s<i 



7V ? 



1 

" Jo 







\ <e( sup 




/ V0<s<ait 





<4E M) 



4airei 



(5.41) 



2a 



< ait. 



Similarly, for some 03 6 (0,oo), we have 



E ( sup 

v0<s<t 



1 y S,l 



1 



tf(l?)T?(u)du 



< a 3 t. 



(5.42) 



Also, by Assumption 2.3 



sup 

0<s<i 



' X ' X* 



Mi n 



n,* * 

Mi Mi 



/x*|6i|t. 



(5.43) 



Next from Lemma 5.2 (see (5.7)), we have in case 1(a), for some 04 £ (0, 00) 

E(V Q+ I SUp \$»?(8)\) < - a * t r . 



(5.44) 



We note that = in cases 1(b), 2 and 3, and so the above estimate (5.44) holds for all 
cases. Finally, we show that for some (15 G (0,oo), 



E ( n-( Q+i ^) 



sup 

0<s<t 



Gf + (s) <a 5 (t + l). 



(5.45) 
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For this, from ( 5.5 ) and the fact that a+^j- > 0, it suffices to show that there exists G (0, oo) 



such that for all n € N, 



E { sup 

y0<s<t 



< a 6 (t + l). 



Using Doob's inequality and a standard martingale argument, we have that 

2 



E { sup 

y0<s<t 



< 4n-(" +1 >E ( |M" +1 



4n^ +1 ) E E (\M) 
l<k< [W+H] 

An-^ J-, E(E(|M fc ™-M^ 1 | 2 |^ 1 

l<k<\n"+ 1 t\ 



, i - |ilC-i| 2 ) +4n-^ +1 )E(|ilC +li | 2 " \M? n „ +H} \ 



+ 4n^ +1 >E ( E f |M£ +lt - M[ nV+H1 , 



< a 7 (t + 1) 



where the last estimate uses the fact that \Mu — M^_-^\ is uniformly bounded in n and k. 
Combining (5.41) - (5.45), we have for some as G (0, oo), 



E ( sup W n {s) ) < a 8 (t + 1). 

y0<s<t 



(5.46) 



0, and so from (5.37) Q n 0. Following the proof of Lemma 



Consequently, W r ' 

(i) (see equations fl5T27| ), (JO^, (j^MJ), and ( |5732[ )), we have T n T* . 
Lemma 5.5. Under the control policy {T n } ; (Q™^, • • • , Q^lK-l) 



5.3 



as n — > oo. 



Proof: Once again all the quantities in the proof are considered under the control policy 
{T n }. Without loss of generality, we assume (cr(l), . . .,a(K)) = (1,2, . . . , K). For t > 0, let 

WH K (t) := J2f=i ^S- It suffices to show that W™ K 0. Define for t > 0, n > 1, 



A?(i) = n~ a A?(nt) 



nt 



n 



nt 



Df(t) = n~ a Df(nt) 



nt 



n 



nt 



From (2.10) and (4.1), we have for t > 0, 



K-l 



K—l \n,* 



l\t + n 



nt 



K-l 

1 - E i <7* 

i=i 



+ E -s? / ( - A i'*) ds-n~ a ^2^ (^(Y s n ) - ft*) T](s)ds 
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Define, for t > 0, 



«"<<> = £ - £ + »- £ -4 /" ( W) - xf) ds 

j=l Pj j=l Pj j=i J 

(K-lyn,* \ t ( K-l \ 



Then for £ > 0, 



wn K (t) = e(t) + e n t + c(t). 



From Lemma [5.3[ ii), we have that £ n converges to a Brownian motion with drift and variance 
a -K> wnere a -K ls gi ven as follows: 

Casel. <^£^. 

j = l «V j=i l<i<j<K-l^^ 

Case 3.^ = 1: 7^ + 2 £ 

Here A is as defined in Lemma 15. II Next observe that 

( n (t)=n 1 a l { Qn (s)=0J=h K _ 1} ds = n l a l{#n if(s ) =0 } ds > 
which is nondecr easing and increases only when W™ K (-) = 0. Define for t > 0, 

r n (i) = sup {0 < s < t : #™ x (s) = o} . 

Since WH K (fl) = 0, r n (t) > 0, a.s. By definition, W™ K {T n {t)-) = 0. Also since W™ K (s) > 
for all s G (r ra (i),f], we have C n ( rn (*)) = C n (0- Therefore, we have 

< W2 K (t) = WH K (t) - W^ K (T n (t)-) = £ n (i) - e(T n (t)-) + n (i - r n (t)), (5.47) 

and so 

- -r^(t - r n (t)) < — [C(t) - C(r n (t)-)} . (5.48) 
From the proof of Lemma 5.4 (see ( |5.41 ), ( |5.42 ), ( 5.44[ ), and (5.45)), we have that 

E ( sup [\C(s)\ + \C(r n (s)-)\}) 0. 
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Also note that 



' K-l 



n 



l-a 



Therefore, we have 

r n (t) — > t in probability and uniformly on compact sets. (5.49) 

Thus £ n (i) — C n ( r ™(^) — ) converges to in probability and uniformly on compact sets. Using 
this observation in (5.48) we now have that 

0"(l(.)-7*(-))=*O, 



where t : [0, oo) — > [0,oo) is the identity map. Finally, from (5.47), we have W™ K =^ 0. The 
result follows. ■ 

Proof of Theorem I4.lt 

As in the proof of Lemma |5.5[ we assume without loss of generality, (c(l), . . . , a{K )) = 
(1,2, ... ,K). Also, once again all the quantities in the proof are considered under control 
policy {T™}. From (5.46) it follows that (5.18) holds with {T ra }. Thus from Lemma 5.3, we 



have X n =>• X, where X is the Brownian motion introduced in Lemma 5.3 Next recall that 

I n as defined in (5.38) satisfies: J"(0 ) 0; !(•) is nondecreasing; and increases only when 

W n (-) = 0. Therefore, from Definition 5.1 we have for t > 0, 



W n (t) = F (J2^\(t). (5.50) 
\i=l Pi J 

Using the Lipschitz continuity property of T (Proposition |5. l[ii) ) , and recalling W* defined in 



(4.6), we have that 



W n => W*. 



Also from Lemma 



5.5 



(Q1, 0, and so from (JK37J), 



(5.51) 



To complete the proof we now establish a suitable uniform integrability estimate. From ( |4.1 
and (5.37), we have 



n,* 



,=1 ^ 



K 
i=l 



(5.52) 



where 



1 

n 



/// 



A?(l?)cfe -iV" 



o 



■li- 



nt 



tf(Y?)i?(s)ds) +n -^ U —^G nV+ (t) 



X? 



X* 
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Next note that 

Ht) = 
< 

and 

Define for t > 0, 
and 



A 



inf y 

i=l 



A 



inf V 



0<s<t 



n,* 
i=l Mi 



K 

*E 



n,* 
,=1 ^ 



jTO „-«+ 
=1 >■*» 



K 



inf y 



" 2 inf 



i=l 



Mi' 



(5.53) 



K 



o<«<^ Mi' °- s -*t? Mi' 

Q](t) = Q](t), j = l,2,...,K-l, 



a 



(5.54) 



Qm = Qm + fit 



II 



1=1 ^* 



i=l 



nT* • 

Mi 



Let for i > 0, 



i=i Mi 



A" 



i=l ^ 
A 



n,* 
i=i Mi 



2 inf 



-$^(Y) 



1=1 1 



A 



(s) 



1 M 



0<s<t^ fj,"' 
i=l 1 



where the last equality uses (5.52). Using the estimates in (5.53) and ( |5.54 ), we observe that 



K 

V] n,*' ~ SU P 
~[ Mi o<s<t 



A 



A 



A 



A 



ME^). 



i=l 



Mi 



i Mi 



0<s<< ' u 
1=1 



re,* 



=1 Mi 



(5.55) 



From (5.41) - (5.43), and (5.45), we have for some a\ G (0,oo) 



supE ( sup \X n (s)\ z ) < ax(t + 1), for all t > 0. 

y0<s<t 



Using this estimate in (5.55) we have, for some a 2 G (0, oo), 



From Lemma 



5.2 



E sup W n {s) < a 2 (t + 1). 

\0<s<t J 

n - a +\^i =>. 0, and so 

Q%=>H* K W*, Q*=>0, j = l,2,...,K-l. 



(5.56) 



By (5.56), we now have that 



E 



e~ 7t (c • Q n {t))dt ) E ( / e^'c^W^t)^ ) = J*. 



7*, 
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Since Q n = Q n in cases 1(b), 2 and 3, we have the result for these cases. Finally consider the 
case 1(a). Using Lemma 5.2 we have for some 0,3,0,4 G (0,oo), 



E 



< 



e-^c • Q n (t)dt ) - E I / e" 7t c • Q n {t)dt 



-7*„ . h n i 



fr^Jo \o<s<t J n v l l 2 



<E(^y e" 7 *c- \Q n (t) -Q n (t)\dt 
e'^tdt 0. 



The result follows. 



Proof of Theorem [371} Let {T n } be a sequence of admissible control policies. The result 
holds trivially if J{{T n }) = liminf^oo J n (T n ) = 00. Suppose now that J{{T n }) < 00. Let 
{n'} be a subsequence such that J n (T n ) — > J_{T n }, as n' — > 00. Henceforth we relabel the 
subsequence {re'} as {n}. Throughout the proof, all quantities are considered under the control 
sequence {T n }. Recalling ( |5.37| ) and that W n (t) > 0; I n is nondecreasing and starts from zero, 
we have from Proposition |5.1| (hi) that for t > 0, 



K 



I n (t) > i n -*(t) := - inf V 



(5.57) 



and with W n '*(t) := Y$Li + I n, *(t) we have for all t > 0, 

W n {t) > W n '*{t). 

Also from Definition j 5 . 1 j , we have W n, *(t) = T (^2^ =1 4^] (t). From Lemma 



(5.58) 



the Lipschitz continuity of T (Proposition |5.1| (ii) ) , we now have 

W n '* W*, 



5.3 'ii), and using 
(5.59) 



where W* is defined by (4.6). 



Next, for fixed w > consider the following linear programming (LP) problem: 

K K 

Minimize ggK if Cj^ subject to = w. (5.60) 

+ i=i i=l Pi 

Elementary calculations show that the the value of the above LP problem is 

V(w) = c cl<K) nl {K) w, (5.61) 

and the corresponding solution is 

Qa{K)= V* a (K) w i and qj = 0, j = a(l) a{K - I) . (5.62) 

Therefore, letting W n {t) = Y* =1 for c > 0, 



K 



c ■ Q n {t) > V(W n (t)) = V(W n (t)) + c K f K J2 Q?(t) 



8=1 



> V{W n {t)) + Z n {t). 
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where once again, without loss of generality we assume, (c(l), . . . , cr(K)) = (1,2,..., K) and 
Z n {t) = Ck il* k YLx Qtm-jfc - & Since J({T™}) < oo, 



lim sup E f e-^\Z n (t)\dt < lim max 

n->oo Jo 0. n ~ >0 ° 1 



1 



n n '* U* 



J n (T n ) = 0, 



where c = minj{cj}. Thus from (5.59) 

lim J n (T n ) = J{{T n }) > liminfE ( [°° e'^V {W n '* (t))dt 

n— >oo n— Yoo \Jq 



>E^ e-^V(W*(t))dt 
= J* = lim J n (T n ), 

n— >oo 



where the last equality follows from Theorem 4.1 



Appendix A: Proof of Lemma 4.1 



We begin by introducing a 1-dimensional control problem referred to as the 'workload control 
problem' 

Definition A.l. The workload control problem (WCP) is to find an M.+ -valued RCLL {Ft}t>o 
adapted stochastic process I which minimizes 



subject to 



E I J i "Vill i /Mr// 



K 



Xj(t) 

!(•) is nondecreasing and 1(0) = 0, 



W(t) = -^T 1 + ht) > 0, for all t > 0, 
i=l ^ 



where X and {J-t}t>o are as * n Definition J^.l and V is defined in (5.61) 



(A.l) 



Proof of Lemma 4.1: Recall (W*,I*), defined in (4.6). From (4.7) we see that, for every 



t > 0, Q*(t) is a solution of the LP problem (5.60) associated with W*(t). Thus 



K 



Y J c i Qm = v{w*{t)). 



(A.2) 



i=i 



Also note that the pair (W*,I*) satisfies the two conditions in (A.l). Furthermore, if (W,I) 



is another pair satisfying (A.l), we have from Proposition 5.1 iii) that Wit) > W*(t) and 



consequently, V(W(t)) > V(W*(t)), for all t > 0, where V is as defined in (5.61). 
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Now let fj & A and define the corresponding Qi, i = 1, • • • K, and / through (4.5). Define 

A 



w(t) = 



Qi{t) 



i=l 



Clearly (W,I) satisfy (A.l) and so from the above discussion 

V{W{t)) > V(W*{t)), for all t > 0. 



(A.3) 



(A.4) 



Also, from the definition of V and flAT3| ) we see that Y^Li^Qii 1 ) > v i w { t ))^ for a11 t > °- 



Combining this inequality with (A. 2) and (A.4) we get that Ylf=i c iQi(t) — J2i^=i c iQ*i (^)> f° r 
all t > 0. Since fj £ A is arbitrary, we have the result. ■ 



Appendix B: Proof of (5.6) 



We only treat the case when K = 1. The general case can be proved similarly. We first consider 
the special case when Y n is stationary. Denote by E^n the expectation operator when Y n (0) 
has distribution vr n . Let M n (t) = ?(n)~5 M£, t ,t > 0. Noting that \AM?\ < ai<?(rt)~^ for all 
t > 0, we have that for any e > and i > 0, 



lim E^n V |AA4?| 2 1 



{|A.M"|>e} 



0. 



v 0<s<t 



By martingale central limit theorem (see Corollary VIII. 3. 24 in [8]), it suffices to show that 
{M. n )t — > — 2t7r(AQA) in probability for all t > 0, where (A4 n ) denotes the predictable quadratic 



variation of the martingale Ai n . We note from (5.4) that 

1 



M n (t) = \ n (Y n (s(n)t)) - A n (y n (0)) - / Q n X n (Y s n )ds . 

v^( n ) V Jo J 



f(fl)t 



Using Lemma VIII. 3. 68 in [8], we have 



(A4 T 



Q n (A n ) 2 (Y s n ) - 2X n Q n X n {Y s n ) ) ds. 
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We then note that 



IEtt" ( {M n ) t + 2tt(XQX) 



^ E 



?(n) 



l<fc<L?(n)iJ 



fc-1 



Q"(A n ) 2 (f s n ) - 2X n Q n X n (Y s n ) + 2vr(AQA) ) ds 



_02_ 



" ^ + ^2 E E -" jf (Q n (\ n )\Y?) - 2X n Q n X n (Y s n ) + 2vr(AQA)) ds 

(Q n (A n ) 2 (y s n ) - 2A n Q™A ri (y™) + 2vr(AQA)) ds 

fex-l V ' 



i<fc<L?(™RI 



E 



l<fci</i2<L?(n)tJ 



a 3 2 



Q n (A n ) 2 (y s n ) - 2X n Q n X n (Y s n ) + 2tt(AQA) ) ds 



s(ri) ?(n) 2 



a 3 2 



E n n (Q n (X n ) 2 (Y s n ) - 2X n Q n X n (Y s n ) + 2~(AQA) ) 

(Q n (X n ) 2 (YD ~ 2X n Q n X n (Y?) + 2tt(AQA)) ds 0£ 

k 2 -i v J 

E w n (Q n {X n f{Y^)-2X n Q n X n {Y^) + 2Ti{XQX)^ds 



l<k 1 <k 2 <\s{n)t\ 



[ k2 (P n ' s - kl (Y£-i, Q n (X n f) ~ 2P n ^(Y^_ l ,X n Q n X n ) + 2tt(XQX)) ds 



Noting that 7r ri (Q n (A ?1 ) 2 ) = and vr n (A n Q n A n ) -)• vr(AQA), and using Assumption 
that for A;2 — 1 < s < fo, 



2.1 



we have 



sup 



pn, a -fci fa Q n( x n ) 2 ) - 2P n ' s ~ k ' (y, X n Q n X n ) + 2tt(AQA) 
Thus we have that 



< a 4 p s ~ kl . 



E{(M n ) t + 2n(XQX)) < ^- + 

1 ' an) s(n) z 



E 

l<Jfci<fc2<|?(n)tJ 



k-z 



fca-l 



— >■ as n — > oo. 

This completes the proof for the case when Y n is stationary. 

Finally we consider the case when Y n (0) has an arbitrary distribution. It suffices to consider 
the setting where E n (0) = x £ L, for all n > 1. The argument below follows the proof of 
Theorem 4.3 in [5]. Define for r £ (0, oo), 
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Clearly, for any r £ (0, oo) 



sup \G n (t) - G n ' r (t)| ^ as n -> oo. 
t>o 



(B.l) 



Let be a real valued bounded Lipschitz function on C([0, oo), M). Then from (B.l), we have, 
as n — >• oo, 

where K x denotes the expectation operator when Y n (0) = x a.s. Using the Markov property, 
we have 

E x [<f>(G n > r )} = [ P n ' r (x,dy)E y [cf ) (G n )}. (B.2) 
Thus we have for some a§ G (0, oo), 

|E^(G n )]-E^(G n )]| 

E 1 .[0(G")]-E a; [0(G^)]+ / (P^(x,dy)-vr™(y))E 2/ [0(G")] 



< |E»[0(G^)] - E a #(G n ' r )]| + a 6 ||P^(x, •) - tt* 



Using (B.2) and Assumption (2.1) we see that the last expression converges to on sending 



first n — > oo and then r — > oo. This completes the proof of (5.6) 



Appendix C: Simulation results in Example 3.1 



We continue with Example 3.1 We plot the total cost and total discounted cost at each discrete 
time. More precisely, recall that the holding cost c = (20, 25)' and discount factor 7 = 2. For 
10 sample paths, define 



10 



and 



10 



C 2 (<) = ^ E (cie-^, s W + Pae^.(t)) . 



Here Cf(t) measures the total cost for the queueing system at time t, while C^it) gives the 
discount cost at time t. 



Let n = 100. We plot Cf(t),t = 0.1, 0.5, 1.0, . . . , 100 and C%(t), t = 0.1, 0.2, 0.3, . . . , 10 for 
six different scalings. In all the following figures, the first plot gives Cf and the second one is 
C2 ■ We use red color to represent c/i*-rule and green color for dynamic c/z-rule. We observe 
that in all the plots, the red line is located below the green one for most of time, which says 
that the cost under c//*-rule is smaller. 
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Fig 2. Average sample path for the case when v = 1 and a — 1/2. 
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